It's 7 AM, I'm sitting on my sofa at home reading the news, and one of our guys out on the road IMs me "Can I call? Problem with the demo." The demo is an important one and it's in 45 minutes, this is what life is like when you're a technology vendor.
The prospect is a new area of business for us, so it's pretty strategic. They sent us a few tens of thousands of data records in XML (yay!) of astounding, remarkable, mind-bending complexity (sigh). There are three kinds of objects in this system and they all have many-to-many relationships with all the others. It took me literally days of programming to actually understand the data structures.
The good news is that once we finally figured out the data, it was easy to build some pretty dramatic maps that bring the obvious problem areas and strategic issues to the top in bright colors that nobody can possibly miss. And then the map drill-down works like a champ.
Except for, 45 minutes before the demo, one of the top-level map items had
areas colored green to say they're OK, others colored red to say they're in
trouble, and the numbers in the labels said the opposite: 17% of
capacity
in lurid pink, 88% of capacity
in soothing
green.
It turns out that a slight last-minute "improvement" to the input data filter by yours truly had, well... Perl is such a great language, except when it's not. One line fix to the data filter, one-button regen of the maps, then all the tedium and protocols of getting it staged to the colo, and the demo went like clockwork, stand by for a press release.
Every management team I've ever been on has believed that you should build processes so that stupid mistakes don't happen and you don't have to pull last-minute rabits out of the hat.
But, software development is tricky and perverse and routes around good management practices.